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ABSTRACT 


This paper summarizes a subset of the Advanced Modeling Simulation and Analysis (AMSA) Capability Roadmap that 
was developed for NASA in 2005. The AMSA Capability Roadmap Team was chartered to “To identify what is needed 
to enhance NASA’s capabilities to produce leading-edge exploration and science missions by improving engineering 
system development, operations, and science understanding through broad application of advanced modeling, 
simulation and analysis techniques.” The AMSA roadmap stressed the need for integration, not just within the science, 
engineering and operations domains themselves, but also across these domains. Here we discuss the roadmap element 
pertaining to integration within the engineering domain, with a particular focus on implications for future observatory 
missions. The AMSA products supporting the system engineering function are mission information, bounds on 
information quality, and system validation guidance. The Engineering roadmap element contains 5 sub-elements: (1) 
Large-Scale Systems Models, (2) Anomalous Behavior Models, (3) advanced Uncertainty Models, (4) Virtual Testing 
Models, and (5) space-based Robotics Manufacture and Servicing Models. 
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1. INTRODUCTION 

In late 2004 NASA established approximately two dozen teams, composed of individuals from government, industry 
and academia, to develop 30-year strategic and capability roadmaps 1 to support the NASA missions in Science, 

Exploration, Space Operations and Aeronautics Research. The completed roadmaps (6 Strategic Roadmaps and 15 

2 

Capability Roadmaps ) were submitted to NASA in May 2005. One of these teams, the NASA Capability Roadmap 
Team for Advanced Modeling, Simulation and Analysis (AMSA) was chartered to “To identify what is needed to 
enhance NASA’s capabilities to produce leading-edge exploration and science missions by improving engineering 
system development, operations, and science understanding through broad application of advanced modeling, 
simulation and analysis techniques.” The context of this charter was to identify capabilities that are needed to be 
available to everyone (NASA itself as well as its industry and university partners) responsible for the design, 
development and operation of NASA missions. The theme of the AMSA roadmap, as illustrated in Figure 1, was greatly 
enhanced integration, not just within the science, engineering and operations domains themselves (horizontal 
integration), but also across these domains (vertical integration). 

The AMSA Capability Roadmap had five major elements: Science Modeling, Operations Modeling, Engineering 
Modeling, Integration, and M&S Environments and Infrastructure. The first three elements deal with capabilities for the 
different types of horizontal integration, the fourth with the vertical integration capabilities, and the fifth with cross- 
cutting capabilities. This paper summarizes the Engineering element of the AMSA Capability Roadmap, with a 
particular focus on implications for future observatory missions. (Note that there was a separate Capability Roadmap 
Team that dealt with Systems Engineering as well as Cost and Risk models; hence, these important engineering topics 
were not within scope of the AMSA Capability Roadmap.) 
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As the 2004-2005 roadmapping results were supplanted by the priorities of the new NASA management, this paper 
should not be construed as representing the official position of NASA, but merely the perspectives circa early 2005 of 
the AMSA Capability Roadmap Team members themselves. 



Figure 1. Integration Theme of the AMSA Capability Roadmap 

2. AMSA CAPABILITY ROADMAP BACKGROUND AND OVERVIEW 

The AMSA Capability Roadmap team was co-chaired by Erik Antonsson and Tamas Gombosi. The team consisted of 
19 members: 7 from NASA, 5 from industry, and the remainder from other government laboratories and industry. A 
complete list of team members and their affiliations (at the time of the roadmapping work) is provided in Appendix A. 
The authors of this paper are the team members that focused on the Engineering element of the roadmap. 

A public workshop was held on November 2005 in Washington DC, at which 17 presentations were given to the co- 
chairs of the teams; an additional 31 white papers were submitted for consideration. The full AMSA team met several 
times from January through May of 2005. To expand the knowledge base of the team, an additional 25 invited speakers 
made presentations to the full AMSA team. In addition the ASMA Capability Roadmap Team had some insight into the 
detailed hardware roadmaps emerging simultaneously from the 12 Capability Roadmap Teams that focused on needed 
developments in instruments, launch vehicles, entry systems, etc. (See, for example, the summary of the Advanced 
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Telescopes and Observatories Capability Roadmap .) The NASA Science Mission Directorate, representing 
Headquarters, provided most of the coordination and direction for the AMSA Capability Roadmap activity. The 
roadmap consequently focused very strongly on capabilities needed for future science (as opposed to exploration 
systems or aeronautics research) missions. Although the submitted roadmap contained some conjectures about 
capabilities for exploration systems or aeronautics research missions, both directorates have undergone dramatic 
replanning activities in the past year. Hence, this paper will mainly focus on portions of the AMSA Capability Roadmap 
that support NASA’s science missions (as they were understood in early 2005). 

The AMSA Capability Roadmap Team was supplied with Design Reference Missions, which in the case of the Science 
Mission Directorate, were the missions envisioned over the next several decades that this roadmap should support. 
Table 1 provides a partial list of these Design Reference Missions. 
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Launch 

Year 

Mission 

2009 

NPOESS Preparatory Project 

2010 

Solar Dynamics Observatory 

2010 

National Polar-orbiting Operational Environmental Satellite System (NPOESS) 

2010 

Laser Interferometer Space Antenna (LISA) 

2013 

Global Tropospheric Wind 

2013 

Mars Sample Return 

2013 

Venus In Situ Explorer (VISE) 

2014 

Solar Orbiter 

2014 

Jupiter Polar Orbiter with Probes (JPOP)/Jupiter Interior Mission (JIM) 

2014 

Terrestrial Planet Finder-Coronograph (TPF-C) 

2014 

Constellation-X 

2015 

Large-Aperture UV Optical Observatory 

2016 

Global Tropospheric Aerosols 

2018 

Total Column Ozone 

2019 

Terrestrial Planet Finder-Interferometer (TPF-I) 

2020 

Geosynchronous Earth Orbit Interferometric Synthetic Aperture Radar 

2020 

Constellation 

2020 

In-Space Construction 

2023 

Ll-Diamond 

2025 

Geosynchronous Global Precipitation 

2025 

Life Finder 

2027 

Titan Sample Return 


Table 1. Representative Science Design Reference Missions for AMSA Capability Roadmap 


3. ENGINEERING MODELING, SIMULATION AND ANALYSIS TODAY 

NASA’s engineering functions span from high-level system analyses to detailed component design and development. 
(Recall that in the context of the AMSA Capability Roadmap, there is a distinct operations element.) Current NASA 
missions are supported by engineering tools that are significantly advanced over those available 20 years ago. This 
increased capability has been enabled not just by the obvious improvements in computer hardware and system software, 
but also by great strides in first-principles-based discipline models and commercial software tools that facilitate large- 
scale integration applications, such as end-to-end system modeling. 

As is apparent from recent and proposed missions, e.g., JWST and TPF, NASA has indeed exploited early design trades 
and extensive modeling and simulation to lower risk before selecting a particular design and the contractors to carry out 
the manufacture and testing of the system. Despite these advances new missions are still straining modeling resources to 
support system engineering decision processes and accurately predict performance. While usually effective for the high- 
risk environment in which NASA operates, this sometimes contributes to dramatic failures, and has also limited the 
development of innovations that would measurably improve our engineering capabilities. And, while all organizations 
wrestle with how to stay competitive in their business, NASA’s engineering modeling capabilities remain discipline 
specific, are too highly dependent on heroism and credibility of key individuals, and have largely fallen from the state- 
of-the-art in a number of key areas relevant to future science missions. 

The singular factor that most contributes to the limitations of today’s system models is complexity -the complexity of the 
systems themselves as well as the complexity of the models of the systems. Although our system models have 
undoubtedly increased in complexity, they have nonetheless not kept pace with the increased complexity of the systems 
themselves (and our confidence in these complex system models has lagged even further behind). The increased system 
complexity is driven by the increased coupling between the constituent subsystems, which has often proven necessary in 
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order to achieve the desired system performance. This system complexity is addressed in the models through end-to-end 
model architectures and increased fidelity in the discipline models. However, there are some serious shortcomings to the 
current practice of model development. 

A persistent challenge is model validation. For one thing, the current practice of developing the model concurrently with 
the hardware typically allows minimal time for validation of the model. This delay in model realization and validation 

translates into a large financial risk since 90% of mission costs are committed within the first 10% of the development 

4 

cycle and changes are very costly . Moreover, full system models are rarely themselves validated; rather isolated 
components of the full system are validated by comparison to specific discipline model results and with subsystem tests 
such as modal testing. The methods of data management and model correlation are not well established and frequently 
rely upon ad hoc procedures. Very little automation exists between testing and model updating. For example, models 
are currently employed to estimate the ambient environment effect and total ionizing dose limitation that spacecraft 
electronics can withstand. These models then drive experiments on physical parts. Although such tests guide 
experiments the results are not currently fed back toward improving the capability of the models. Both empirical and 
analytical methods must be improved to address this aspect of model validation. Finally additional limitations of current 
modeling are the lack of dynamic coupling between the performance models and science models and programmatic 
issues such as cost and schedule. This means there is little opportunity to fully exercise these models and take advantage 
of the cost savings and reduced mission risks afforded by exploration of the trade space and optimization enabled by 
more capable environments. 

A related problem is that large uncertainties are associated with most system models. As an example, certain aspects of 
the space environment (e.g., Mars atmosphere) are so poorly modeled (or inherently variable) that large design margins 
must be carried. Currently the instrument design process is managed by error budgets, which are evaluated by 
covariance propagation models. However, consistent methodologies and approaches and the ability to handle large-scale 
systems has not been firmly established and varies significantly. These shortcomings can be cast into the larger arena of 
uncertainty and margin management. Quantifying uncertainties and linking these to assessments of risks via 
probabilistic margin assessment needs to become part of the standard practice, but the mathematical foundations are 
weak in some areas and current tools are inadequate. 

There are certainly other system-level limitations facing NASA and industry. One is the lack of a unified framework for 
end-to-end system models that not only facilitates the integration of the various discipline models at all needed levels of 
fidelity, but also the capturing of the uncertainty, analysis/verification, and validation processes for the discipline and 
system models. Integrated full-breadth modeling capabilities are rare except for very low fidelity and largely empirical 
cases. Data flow between commercial design, and analysis packages has been amply demonstrated, but the process is far 
from seamless and often requires human intervention and development of specialized data translators. Models of 
advanced technologies that can be used in system-level assessments for technology investments are weak. 

At the discipline level, model fidelity is already adequate in some areas, e.g., deep space navigation, for design 
decisions on complex systems. But in many other areas, however, this is not the case. For instance, NASA has a 
capability to model portions of a spacecraft/lander electromagnetic interference environment. Full design optimization 
of antenna multipath and near- field EM interference on the complete spacecraft with realistic geometry, however, is not 
currently possible due to issues of model maturity compared to recent advances in physics, lack of engineering model 
integration, limitations of desktop-based analysis tools, and a poor ability to perform iterative high fidelity design trade- 
space exploration based on a seamless concept-to-flight modeling and simulation analysis capability. Such limitations 
are general in nature and apply equally well to many other engineering disciplines. 

A common thread to many of these obstacles is that since there is little or no commercial market for many of the tools 
that NASA needs for its missions, development of specialized capabilities must necessarily be sponsored by NASA. 

4. FUTURE ROLE OF ENGINEERING MODELING, SIMULATION AND ANALYSIS 

The AMSA Capability Roadmap provided three options for investment. Level 1 dealt with improving these discipline- 
modeling capabilities that were most critical to the Design Reference Missions. Level 2 addressed horizontal, cross- 
discipline, integration in the Engineering domain, recognizing that many of the future missions proposed in the Design 
Reference Missions set will require integration of extremely complex technologies, which will challenge management 
and system engineering. Level 3 focused on vertical integration — across Science, Operations and Engineering. In this 
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paper we summarize the Level 2 Engineering element of the AMSA Capability Roadmap, which had the following five 
sub-elements: 

• Large-scale systems models (LSSMs), which enable system evaluations and therefore leverage the increased 
knowledge gained early in the design cycle. These are intended to be evolutionary cradle-to-grave tools with an 
environment supporting data management and multiple optimization tools, allowing full exploration of the 
trade space. These are characterized as multilevel models, with an open but controlled architecture that allows 
distributed resources and computing. 

• Anomalous behavior models (ABMs) for proactive consideration of low-probability but high-risk events. 
These models, which are today typically reserved for post-mortem investigations, should become more 
embedded in the early design cycle process, thereby minimizing failure modes and effects. It is proposed that 
artificial intelligence tools play a larger role in evaluating system culpability by developing Al-based “agents- 
of-doom” software tools. 

• Increased support and rigor in development of uncertainty models (UMs). These are tools to characterize 
inherent variability due to lack of knowledge and errors and are strongly coupled to design space size and 
optimization processes. In the early phases, these are important for characterizing chances of mission success. 

• Selective use of virtual testing models (VTMs) due to environmental and economic constraints. This is the use 
of modeling for the untestable product and/or unobservable parameter and for updating flight LSSM. 

• Support for increased space-based robotics manufacture and servicing models (RMSM). This is a virtual 
environment for dynamically replicating assembly, servicing, and repair processes in space. 

In Figure 2 we show a functional representation of this concept. (Note that in accordance with the vertical integration 
theme of the roadmap, we include the linkage to the Science and Operations models.) Managing this complexity will 
require full integration of performance, science, and cost models within an environment which facilitates data 
management, optimization, and distributed computational and user interaction - this is the domain covered by LSSM. 



Figure 2. Functional diagram of proposed Large-Scale-System Modeling to support future missions. 
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To establish the validity of these models, separate tools are needed to establish uncertainty bounds on discipline and 
system models. There are many available frameworks that can be borrowed from different communities that should be 
better established in NASA’s modeling tools. For example, the control community has developed formalisms known 
under the generic term of robust control (p analysis, H°° control), which deal with modeling uncertainty. The statistics 
community has evolved new tools based upon Bayesian techniques utilizing efficient Monte Carlo Markov Chain 
methods, which can help evaluate results of complex systems. The Department of Energy’s Accelerated Strategic 
Computing program has made significant progress recently on uncertainty quantification for AMS A. These tools could 
be used now on complex systems such as JWST and certainly would reduce risk on future missions. However, there are 
many challenges still remaining. 

To increase the effectiveness of these tools, we propose earlier incorporation of the system modeling activities. This 
follows a recent trend by NASA to conduct multiple studies of architecture, requirements flowdown and cost in Pre- 
Phase A and Phase A studies. A diagram illustrating the temporal aspects from initial mission definition to on-orbit 
operations is shown in Figure 3. A rough outline of a developmental timeline to reach these goals for integration of 
these tools into a coherent framework is shown in Table 2. The linkage of Level 2 of the Engineering element of the 
AMS A Capability Roadmap to some Design Reference Missions is shown in Figure 4 and Figure 5. 


CURRENT 
MISSIONS 
Complexity in- 
creasing, sub- 
system coupling, 
ground testing 
constraints 
(environmental 
programmatics) 

FUTURE 
MISSIONS 
Large-scale 
integration, 
cradle-to-grave, 
rapid proto- 
typing, multiple 
models, expert 
systems, uncer- 
tainty bounds, 
distributed 
computing, 
anomalous 
behavior models, 
model driven 
testing 
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and model updating 


Virtual test models 


Limited/ 

targeted 

system 

tests 


1 

1 

1 

Hi fidelity, ■ 


optimized 

Update 

performance 

flight model 
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1 
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1 
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1 
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1 

1 

1 
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u - Targeting/ 
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• Mid-course 
corrections 

• Anomaly 
resolution 

• Data 
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• Servicing/ln-space 
construction 


Operations models 


Figure 3. Conceptual diagram of early incorporation of system models into the engineering process. 
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Engineering 

Today’s Capability 

2010-2015 

2016-2020 

2021-2035 

Large-scale system 
modeling 

Bucket-brigade 
Developing integrated system 
modeling, significant discipline 
modeling & optimization, 
approximate models. (TRL=3) 

Cradle-to-grave models, rapid 
model deployment, 1st- 
generation embedded data 
management, integrated cost 
models, selected advanced 
discipline models, MDO. 

Seamless model evolution 
through design phases, 
integrated risk models, 
design traceability, additional 
advanced discipline models, 
agent-based. 

Distributed, MDO, environment for 
optimization, advanced data 
management, cost/risk integrated, 
science and operations, cradle-to- 
grave models, rapid prototyping, 
integration of other model types. 

Virtual test 
environment 

Widely varies, not baseline 
approach. Fit tool for 
manufacturing. (TRL=3,4) 


Robotic optical assembly 
and alignment. 

Expansive HWIL, max modeling/ min 
testing, auto sys ID/ model update, 
order of magnitude reduction l&T. 

Uncertainty models 

Probabilistic uncertainty 
propagation tools. Some 
uncertainty characterization. 
(TRL=3-6) 

Non-probabilistic 
uncertainty tools. Expanded 
uncertainty characterization. 

Tools for rigorous 
uncertainty bounds in the 
validation domain. 

Tools for rigorous uncertainty bounds 
in the predictive domain. Input 
uncertainties fully characterized. 

Anomalous behavior 
models 

Typical using current models 
with some add-ons as mishap 
investigation. (TRL=2-5) 

Subsystem Al agent of 
doom. High-fidelity abort 
and damage analysis. 

Full system Al agent 
of doom. 

Explore full failure/ anomaly mode space 
during design. Al agent of 
doom. Real-time isolation and 
resolution. 

Robotics mnfg/ servcg 
(MS) models 

Commercial mainly, minimal 
space-based modeling 
(servicing). (TRL=3-6) 

Human exploration hazard 
models. 

Robotic optical assembly 
and alignment. 

Virtual toolset enabling dynamic 
assessment of designs for 
space/planetary-based MS. 

Visualization 

technology 

3-D, small-scale dynamic 
visualization, single discipline 
analysis. 

Multidiscipline analysis, 
design space exploration. 

Interactive design steering, 
design space exploration 
agents. 

Hologram, instant visualization 
of dynamic events at multiple scales. 


Table 2. Rough outline of plan to develop additional modeling resources for mission support. 

5. CONCLUDING REMARKS 

This paper has presented the perspective of the members of the Advanced Modeling, Simulation and Analysis 
Capability Roadmap Team on the AMSA needs in the engineering domain to enhance the performance, reduce the cost, 
reduce the risk and thereby increase mission assurance for future science missions. We find it timely that a recent 
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National Science Foundation Report has identified a national need for increased capability in AMSA (Simulation- 
Based Engineering Science in the NSF report terms) that is far broader than the NASA science mission need that was 
the focus of our roadmap. Two of the three major recommendations of the NSF report are particularly germane here: 

2. “Formidable challenges stand in the way of progress in SBES research. These challenges involve resolving 
open problems associated with multiscale and multi-physics modeling, real-time integration of simulation methods with 
measurement systems, model validation and verification, handling large data, and visualization. ...” 

3. “There is strong evidence that our nation’s leadership in computational engineering and science, particularly 
in areas key to Simulation- Based Engineering Science, is rapidly eroding. ...” 
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Figure 4. Near-Term AMSA Engineering Capability Roadmap 



Figure 5. Far-Term AMSA Engineering Capability Roadmap 
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